Search for: All records

Automated grading systems, or auto-graders, have become ubiquitous in programming education, and the way they generate feedback has become increasingly automated as well. However, there is insufficient evidence regarding auto-grader feedback’s effectiveness in improving student learning outcomes, in a way that differentiates students who utilized the feedback and students who did not. In this study, we fill this critical gap. Specifically, we analyze students’ interactions with auto-graders in an introductory Python programming course, offered at five community colleges in the United States. Our results show that students checking the feedback more frequently tend to get higher scores from their programming assignments overall. Our results also show that a submission that follows a student checking the feedback tends to receive a higher score than a submission that follows a student ignoring the feedback. Our results provide evidence on auto-grader feedback’s effectiveness, encourage their increased utilization, and call for future work to continue their evaluation in this age of automation. 

Instructors in computer science classes often need to decide between having students use real programming tools to provide practical experience and presenting them with simpler educational interfaces to reduce their cognitive load. Our work investigates the trade-offs between these approaches, by comparing student learning from two offerings of an introductory Python class across several community colleges in the U.S. In the first offering ($N = 219$), students used a real IDE (Visual Studio Code) throughout the entire course. In the second offering ($N = 166$), students used a simplified in-browser code editor, with no setup, for the first three modules and transitioned to Visual Studio Code in the subsequent modules. Our results showed that the second offering led to better learning than the first offering in the first three modules with the in-browser code editor. Moreover, students in both offerings performed similarly in a subsequent module in which they performed local development with Visual Studio Code, suggesting that the ability to use a real IDE was not harmed by the initial use of the in-browser code editor. In addition, we found that students in both offerings improved in their levels of self-efficacy with the course's learning objectives at the end of the class. Finally, we identified that the revisions made in the second offering benefited full-time students more than part-time students. We conclude with a discussion of the trade-offs between employing realistic programming tools and simplified coding environments, as well as suggestions for making introductory computer science classes more effective and accessible.